Operations in a three dimensional display system

ABSTRACT

System and method for invoking 2D and 3D operational modes of a 3D pointing device in a 3D presentation system. A 3D stereoscopic scene and a 2-dimensional (2D) scene are displayed concurrently via at least one stereoscopic display device. A current cursor position is determined based on a 6 degree of freedom 3D pointing device. The cursor is displayed concurrent with the 3D stereoscopic scene and the 2D scene, where the cursor operates in a 2D mode in response being inside a specified volume, where, in the 2D mode, the cursor is usable to interact with the 2D scene, and where the cursor operates in a 3D mode in response to being outside the specified volume, where, in the 3D mode, the cursor is usable to interact with the 3D stereoscopic scene.

PRIORITY DATA

This application also claims benefit of priority of U.S. provisionalapplication Ser. No. 61/663,917, titled “Operations in a ThreeDimensional Display System”, filed Jun. 25, 2012, whose inventors wereJonathan J. Hosenpud and Scott M. Dolim, which is hereby incorporated byreference in its entirety as though fully and completely set forthherein.

FIELD OF THE INVENTION

The present invention relates to the field of three dimensionalgraphics, and more particularly to pointing device, e.g., stylus,operations in a three dimensional display system.

DESCRIPTION OF THE RELATED ART

Three dimensional (3D) capable computing devices and real-timecomputer-generated 3D computer graphics have been a popular area ofcomputer science for the past few decades, with innovations in visual,audio, and tactile systems. Much of the research in this area hasproduced hardware and software products that are specifically designedto generate greater realism and more natural computer-human interfaces.These innovations have significantly enhanced and simplified the user'scomputing experience.

However, additional tools and improvements to the realm of 3D systemsare desired.

SUMMARY OF THE INVENTION

Various embodiments are presented of a system and method for userinteraction with objects in a three dimensional (3D) display system,e.g., a stereo imaging system.

A 3-dimensional (3D) stereoscopic scene may be displayed via at leastone stereoscopic display device. In other words, the 3D stereoscopicscene may be displayed in a stereoscopic manner on the at least onestereoscopic display device. Note that this is not equivalent todisplaying a 3D scene, e.g., a computer generated scene using 3Dmodeling, in 2D on a 3D television operating in 2D mode.

A 2-dimensional (2D) scene may be displayed via the at least onestereoscopic display device concurrent with display of the 3Dstereoscopic scene, where the entire 2D scene is displayed at zeroparallax, i.e., in the plane of the display device's screen. In oneexemplary embodiment, the 2D scene may be a graphical user interface(GUI) that includes one or more GUI elements. In another exemplaryembodiment, the 2D scene may be a background or backdrop for the 3Dstereoscopic scene with one or more 2D objects. In other embodiments,the 2D scene may be any type of 2D scene as desired, includingcombinations of the above.

A current position of a cursor may be determined with respect to the atleast one stereoscopic display device based on a position of a 6 degreeof freedom (6DOF) 3D pointing device. In other words, the cursorposition may be based on the position of a 6DOF 3D pointing device inrelation to the at least one stereoscopic display device (e.g., screen),such as a (physical) stylus. For example, the cursor may be anassociated 3D cursor corresponding to the tip of the physical stylus. Inan alternate embodiment or configuration, the cursor may be positionedat or may correspond to a point offset from the tip. In one embodiment,in being based on the position of the 6DOF 3D pointing device, thecurrent position of the cursor may be based on a virtual beam extendingfrom the position of the 6DOF 3D pointing device. Moreover, the virtualbeam may be a configurable length, e.g., the offset from the 6DOF 3Dpointing device to the cursor may be adjustable. The distal end of thevirtual beam (i.e., the end of the beam furthest from the 6DOF 3Dpointing device) may be or include a virtual stylus tip, and the cursormay correspond to the virtual stylus tip.

The cursor may be displayed via the at least one stereoscopic displaydevice concurrent with display of the 3D stereoscopic scene and the 2Dscene. The cursor may operate in a 2D mode in response to the currentposition being inside a specified volume proximate to the at least onestereoscopic display device. Moreover, in the 2D mode, the cursor may beusable to interact with the 2D scene. Similarly, the cursor may operatein a 3D mode in response to the current position being outside thespecified volume proximate to the at least one stereoscopic displaydevice, where in the 3D mode, the cursor may be usable to interact withthe 3D stereoscopic scene. In some embodiments, in the 2D mode thecursor may be displayed in the 2D scene with a first representation, andin the 3D mode the cursor may be displayed in the 3D stereoscopic scenewith a second representation, where the second representation isdifferent from the first representation.

In some embodiments, the 2D scene may be or include a graphical userinterface (GUI) that includes one or more 2D user interface (UI)elements. In being usable to interact with the 2D scene, the cursor maybe usable to interact with the one or more 2D UI elements. Thus, in anembodiment where the 2D scene is or includes a GUI with one or more 2DGUI elements, in response to the current position being inside thespecified volume proximate to the at least one stereoscopic displaydevice, input regarding a displayed 2D (graphical) user interface (UI)element may be received. In other words, if the current position iswithin a specified volume of space, e.g., adjacent to the display (e.g.,screen), pointing device operations (with or of the 6DOF 3D pointingdevice) may pertain to a 2D GUI element, such as a menu. In oneexemplary embodiment, the volume may be specified or bounded by a planeparallel to the display (screen) and the screen, such that if thecurrent position is between the plane and the display, input from the6DOF 3D pointing device, e.g., stylus (or other pointing device), may bedirected to the 2D UI element, e.g., a menu item, i.e., the specifiedvolume may be defined by a plane parallel to a screen of the at leastone stereoscopic display device, where the specified volume is on thescreen side of the plane.

For example, in one exemplary embodiment the current position may bebelow such a plane, and thus, within the specified volume proximate(i.e., adjacent or near) to the display (screen). In some embodiments,when the current position is in this specified volume, the cursor maychange its appearance to reflect a 2D operational mode. Moreover, in oneembodiment, when in the 2D mode, the cursor may “pop” to the 2D surfaceof the display, and thus may operate like a standard 2D mouse cursor,and may facilitate user interactions with 2D GUI elements or 2D objectsdisplayed in the 2D scene. Said another way, when in 2D mode, the cursormay be displayed as a 2D element on the display, e.g., 2D cursor, ratherthan a 3D cursor in the space above (or below) the surface of thedisplay. Conversely, when the current position of cursor (whose positionmay be computed, but not displayed, since the cursor is in 2D mode, andthus displayed on the surface of the display) leaves the specifiedvolume, the cursor may revert to its 3D appearance and position.

In some embodiments, the specified volume may be defined by respectivebounding volumes proximate to each of one or more 2D user interface (UI)elements on the at least one stereoscopic display device. Alternatively,in some embodiments, the specified volume may be defined by exclusion ofa respective bounding volume of at least one object in the 3Dstereoscopic scene. In other words, the specified volume demarcating the2D and 3D modes of operation may be the volume or space outside thevolume or space around one or more 3D objects in the 3D stereoscopicscene, e.g., outside convex hulls associated with each 3D object.

In some embodiments, the 2D UI element(s) may not be displayed unlessthe 3D pointing device is operating in 2D mode, i.e., the 2D UI may onlybe displayed when the user positions the 6DOF 3D pointing device orassociated cursor within the specified volume. In yet anotherembodiment, the 2D UI element(s) may displayed in a partial orintermediate manner until the 6DOF 3D pointing device or cursor isoperating in 2D mode, i.e., the 2D UI may only be fully displayed whenthe user positions the 6DOF 3D pointing device or associated cursorwithin the specified volume. For example, the 2D UI element(s) may besemi-transparent, or “grayed out” until the 2D mode is active. Ofcourse, in various embodiments, any other type of partial orintermediate display techniques may be used as desired.

In one embodiment, rather than distance from the stereoscopic displaydevice, the orientation of the 6DOF 3D pointing device or cursor, e.g.,the angle of the positioning of the stylus and/or its cursor beam, maydictate or invoke the change to 2D menu mode. Of course, in otherembodiments, any degrees of freedom may be used as desired.

Similarly, as discussed above, the cursor may operate in 3D mode inresponse to the current position being outside the specified volume, andso in some embodiments, input regarding the 3D scene may be received viathe 6DOF 3D pointing device or associated cursor. In some embodiments,the 3D stereoscopic scene may include at least one 3D graphical object,and in being usable to interact with the 3D stereoscopic scene, thecursor may be usable to interact with the at least one 3D graphicalobject. In other words, if the current position is not in the specifiedvolume (e.g., proximate to the display), the 3D pointing device (e.g.,stylus) may operate in a 3D mode, and thus may be used to interact withobjects in the 3D scene, e.g., the house.

For example, in one exemplary embodiment, the current position (e.g., ofthe stylus or its associated 3D cursor) may be above the plane and thusmay operate in 3D mode, e.g., allowing user interaction with respect tothe house. Note that in this embodiment, the cursor may revert to its 3Drepresentation, e.g., 3D cursor.

As discussed above, in some embodiments, the (3D) cursor may bepositioned at or near the tip of the 6DOF 3D pointing device. However,in some embodiments, the 3D pointing device may be capable of extendingthe 3D cursor in a specified direction, e.g., in the form of a ray orbeam of configurable length, e.g., in one exemplary embodiment thecursor may be shown extended from the tip of the stylus, e.g., may be a3D extended cursor. Thus, it may be the case that the stylus tip (e.g.,position of the 3D pointing device) may be outside the specified volume(e.g., above the plane) while the 3D cursor may be positioned within thespecified volume (e.g., below the plane). Accordingly, in someembodiments, the particular position used may be configurable, or maydefault to either. For example, in one embodiment, the current positionmay always be that of the 3D cursor associated with the 3D pointingdevice.

It should be noted that in some embodiments, the specified volume mayhave different shapes or arrangements. For example, as noted above, insome embodiments, the volume may be defined by one or more boundingvolumes, e.g., rectangular solids, spheres, convex hulls, etc.,containing respective objects of the 3D scene, where the specifiedvolume proximate to the display may be the space excluded by thesevolumes. In other words, the specified volume proximate to the displaymay be defined by not being a volume specified for 3D mode. Said anotherway, in some embodiments, the “2D mode” volume may be defined bydefining the “3D mode” volume, where the “2D mode” volume does notinclude the “3D mode” volume. In other embodiments, the specified volumemay include respective volumes defined around (or above) 2D UI elements,such that if the current position is within any of these volumes, the 2Dmode is operational. In one embodiment, the specified volume and thespace outside the specified volume may define respective regions, whereone region is displayed as inactive when the cursor is in the otherregion.

Thus, in various embodiments, the cursor (and/or the 6DOF 3D pointingdevice), or more generally, the 3D presentation system, may operate indifferent modes, e.g., 2D vs. 3D, based on proximity of the cursor tothe at least one stereoscopic display device, or more generally, basedon positioning the 3D pointing device or its associated 3D cursor withina specified volume (or alternatively, outside another specified volume).

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the preferred embodiment is consideredin conjunction with the following drawings, in which:

FIGS. 1 and 2 illustrate exemplary systems configured to implementvarious embodiments;

FIGS. 3A and 3B are flowcharts of a method for invoking 2D and 3Doperational modes in a 3D presentation system based on proximity to adisplay, according to some embodiments;

FIGS. 4A-4C illustrate exemplary operations modes of a 3D pointingdevice per the method of FIGS. 3A and 3B, according to some embodiments;

FIG. 5 is flowchart of a method for user interactions in a 3Dpresentation system using a 3D pointing device, according to someembodiments;

FIGS. 6A-6C illustrate exemplary user interactions in a 3D scene per themethod of FIG. 5, according to some embodiments;

FIG. 7 is a flowchart diagram illustrating one embodiment of a methodfor identifying objects in a 3D scene, according to some embodiments;and

FIGS. 8A-8C illustrate exemplary identification of objects in a 3D sceneper the method of FIG. 7, according to some embodiments.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE EMBODIMENTS Incorporation by Reference

The following references are hereby incorporated by reference in theirentirety as though fully and completely set forth herein:

U.S. provisional application Ser. No. 61/663,917, titled “Operations ina Three Dimensional Display System”, filed Jun. 25, 2012, filed Jun. 25,2012.

U.S. provisional application Ser. No. 61/361,081 titled “User InterfaceElements for use within a Three Dimensional Scene” filed Jul. 2, 2010.

U.S. application Ser. No. 13/174,448, titled “User Interface Elementsfor use within a Three Dimensional Scene” filed Jun. 30, 2011.

U.S. provisional application Ser. No. 61/364,277, titled “Tools for usewithin a Three Dimensional Scene”, filed Jul. 14, 2010.

U.S. application Ser. No. 13/182,305, titled “Tools for use within aThree Dimensional Scene” filed Jul. 13, 2011.

U.S. provisional application Ser. No. 61/561,733, titled “Pre ButtonEvent Stylus Position”, filed Nov. 18, 2011.

U.S. patent application Ser. No. 11/098,681 (U.S. Patent Publication No.2005/0219694), titled “Horizontal Perspective Display”, filed on Apr. 4,2005.

U.S. patent application Ser. No. 11/141,649 (U.S. Patent Publication No.2005/0264858), titled “Multi-plane Horizontal Perspective Display”,filed on May 31, 2005.

U.S. patent application Ser. No. 12/797,958, titled “Presenting a Viewwithin a Three Dimensional Scene”, filed on Jun. 10, 2010, whoseinventors are Michael A. Vesely and Alan S. Gray.

U.S. patent application Ser. No. 13/019,384, titled “ModifyingPerspective of Stereoscopic Images Based on Changes in User Viewpoint”,filed on Feb. 2, 2011, whose inventors are Michael A. Vesely, Nancy L.Clemens, and Alan S. Gray.

Terms

The following is a glossary of terms used in the present application:

Memory Medium—any of various types of memory devices or storage devices.The term “memory medium” is intended to include an installation medium,e.g., a CD-ROM, floppy disks 104, or tape device; a computer systemmemory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM,Rambus RAM, etc.; a non-volatile memory such as a Flash, magnetic media,e.g., a hard drive, or optical storage; registers, or other similartypes of memory elements, etc. The memory medium may comprise othertypes of memory as well or combinations thereof. In addition, the memorymedium may be located in a first computer in which the programs areexecuted, or may be located in a second different computer whichconnects to the first computer over a network, such as the Internet. Inthe latter instance, the second computer may provide programinstructions to the first computer for execution. The term “memorymedium” may include two or more memory mediums which may reside indifferent locations, e.g., in different computers that are connectedover a network.

Carrier Medium—a memory medium as described above, as well as a physicaltransmission medium, such as a bus, network, and/or other physicaltransmission medium that conveys signals such as electrical,electromagnetic, or digital signals.

Computer System—any of various types of computing or processing systems,including a personal computer system (PC), mainframe computer system,workstation, network appliance, Internet appliance, personal digitalassistant (PDA), television system, grid computing system, or otherdevice or combinations of devices. In general, the term “computersystem” can be broadly defined to encompass any device (or combinationof devices) having at least one processor that executes instructionsfrom a memory medium.

Viewpoint—this term has the full extent of its ordinary meaning in thefield of computer graphics/cameras. For example, the term “viewpoint”may refer to a single point of view (e.g., for a single eye) or a pairof points of view (e.g., for a pair of eyes). Thus, viewpoint may referto the view from a single eye, or may refer to the two points of viewfrom a pair of eyes. A “single viewpoint” may specify that the viewpointrefers to only a single point of view and a “paired viewpoint” or“stereoscopic viewpoint” may specify that the viewpoint refers to twopoints of view (and not one). Where the viewpoint is that of a user,this viewpoint may be referred to as an eyepoint (see below). The term“virtual viewpoint” refers to a viewpoint from within a virtualrepresentation or 3D scene.

Eyepoint—the physical of a single eye or a pair of eyes. A viewpointabove may correspond to the eyepoint of a person. For example, aperson's eyepoint has a corresponding viewpoint.

Vertical Perspective—a perspective which is rendered for a viewpointwhich is substantially perpendicular to the display surface.“Substantially perpendicular” may refer to 90 degrees or variationsthereof, such as 89 and 91 degrees, 85-95 degrees, or any variationwhich does not cause noticeable distortion of the rendered scene (e.g.,which sustains a normalized perspective of the user to the normalizeddisplay surface). A vertical perspective may be a central perspective,e.g., having a central vanishing point. In a vertical perspective, theviewpoint may have a line of site parallel to the ground plane (e.g.,floor) and towards a display surface that is perpendicular to the groundplane. As used herein, a vertical perspective may apply to a singleimage or a stereoscopic image. When used with respect to a stereoscopicimage (e.g., presenting a stereoscopic image according to a verticalperspective), each image of the stereoscopic image may be presentedaccording to the vertical perspective, but with differing singleviewpoints. The term “perpendicular perspective” may also refer to thedefinition above.

Horizontal Perspective—a perspective which is rendered from a viewpointwhich is not perpendicular to the display surface. More particularly,the term “horizontal perspective” refers to a perspective which isrendered using a substantially 45 degree angled render plane inreference to the corresponding viewpoint. The rendering may be intendedfor a display which may be positioned horizontally (e.g., parallel to aground plane, e.g. table or floor) in reference to a standing viewpointperspective. “Substantially 45 degrees” may refer to 45 degrees orvariations thereof, such as 44 and 46 degrees, 40-50 degrees, or anyvariation which may cause minimal distortion of the rendered scene(e.g., which sustains the appropriate angled projection of the imagewithin the rendered scene). In a horizontal perspective, a displaysurface may be parallel to the ground plane, but may be some angle offparallel to the ground plane in either the horizontal or verticaldirection. As used herein, a horizontal perspective may apply to asingle image or a stereoscopic image. When used with respect to astereoscopic image (e.g., presenting a stereoscopic image according to ahorizontal perspective), each image of the stereoscopic image may bepresented according to the horizontal perspective, but with differingsingle viewpoints. The term “oblique perspective” may also refer to thedefinition above.

Automatically—refers to an action or operation performed by a computersystem (e.g., software executed by the computer system), without userinput directly specifying or performing the action or operation. Thusthe term “automatically” is in contrast to an operation being manuallyperformed or specified by the user, where the user provides input todirectly perform the operation. An automatic procedure may be initiatedby input provided by the user, but the subsequent actions that areperformed “automatically” are not specified by the user, i.e., are notperformed “manually”, where the user specifies each action to perform.For example, a user filling out an electronic form by selecting eachfield and providing input specifying information (e.g., by typinginformation, selecting check boxes, radio selections, etc.) is fillingout the form manually, even though the computer system must update theform in response to the user actions. The form may be automaticallyfilled out by the computer system where the computer system (e.g.,software executing on the computer system) analyzes the fields of theform and fills in the form without any user input specifying the answersto the fields. As indicated above, the user may invoke the automaticfilling of the form, but is not involved in the actual filling of theform (e.g., the user is not manually specifying answers to fields butrather they are being automatically completed). The presentspecification provides various examples of operations beingautomatically performed in response to actions the user has taken.

Zero Parallax—refers to a display plane coincident with a screen of adisplay device. For example, a 2D scene is displayed at zero parallax.In other words, zero parallax refers to a virtual plane that representsthe display screen.

FIGS. 1 and 2—Exemplary Systems

FIGS. 1 and 2 illustrate exemplary systems which are configured toperform various embodiments described below.

In the embodiment of FIG. 1, computer system 100 may include chassis110, display 150A and display 150B (which may collectively be referredto as display 150 or “at least one display” 150), keyboard 120, mouse125, a 3D pointing device 130, in this exemplary case, a stylus, andglasses 140. In one embodiment, at least one of the displays 150A and150B is a stereoscopic display. For example, in one embodiment, both ofthe displays 150A and 150B are stereoscopic displays. The displays 150Aand 150B may be closely positioned to each other, e.g., where they abut.The angle formed between the displays may be any of various angles,e.g., 90 degrees, 110 degrees, etc.

It should be noted that as used herein, a “3D pointing device” may beany type of pointing device or manipulator capable of indicating atleast a position in 3-space, e.g., x, y, and z, i.e., having at least 3degrees of freedom (DOFs), and in preferred embodiments, has 4 or moreDOFs, e.g., 6 DOF, such as position and orientation with respect to 3spatial axes or dimensions, e.g., x, y, z, (for position) and pitch,roll, and yaw (for orientation). In other embodiments, the 3D pointingdevice may have any combinations of 4 or more DOFs, as desired.

The chassis 110 may include various computer components such asprocessors, memory mediums (e.g., RAM, ROM, hard drives, etc.), graphicscircuitry, audio circuitry, and other circuitry for performing computertasks, such as those described herein. The at least one memory mediummay store one or more computer programs or software components accordingto various embodiments of the present invention. For example, the memorymedium may store one or more graphics engines which are executable toperform the methods described herein. The memory medium may also storedata (e.g., a computer model) representing a virtual space, which may beused for projecting a 3D scene of the virtual space via the display(s)150. The memory medium may further store software for presenting thevarious user interface elements described herein. Additionally, thememory medium may store operating system software, as well as othersoftware for operation of the computer system. Various embodimentsfurther include receiving or storing instructions and/or dataimplemented in accordance with the foregoing description upon a carriermedium.

As indicated above, the computer system 100 may be configured to displaya three dimensional (3D) scene (e.g., via stereoscopic images) using thedisplay 150A and/or the display 150B. The computer system 100 may alsobe configured to display or present user interface elements, e.g.,within the 3D scene, and/or independently of the 3D scene, e.g., 2Dmenus, controls, etc., e.g., behind, under, or to the side of the 3Dscene, using the display 150A, the display 150B, and/or another display,as described in more detail below.

It should be noted that the embodiment of FIG. 1 is exemplary only, andother numbers of displays are envisioned. For example, the computersystem 100 may include only a single display or more than two displays,or the displays may be arranged in different manners than shown. In thisparticular embodiment, the display 150A is configured as a verticaldisplay (which is perpendicular to a user's line of sight) and thedisplay 150B is configured as a horizontal display (which is parallel oroblique to a user's line of sight). The vertical display 150A may beused (e.g., via instructions sent by a graphics engine executing in thechassis 110) to provide images which are presented according to avertical (or central) perspective and the display 150B may be used(e.g., via instructions sent by a graphics engine executing in thechassis 110) to provide images which are presented according to ahorizontal perspective. Descriptions of horizontal and verticalperspectives are provided in more detail below. Additionally, while thedisplays 150 are shown as flat panel displays, they may be any type ofsystem which is capable of displaying images, e.g., projection systems.

Either or both of the displays 150A and 150B may present (display)stereoscopic images for viewing by the user, i.e., objects in a 3D sceneimaged in stereo. By presenting stereoscopic images, the display(s) 150may present a 3D scene for the user. This 3D scene may be referred to asan illusion since the actual provided images are 2D, but the scene isconveyed in 3D via the user's interpretation of the provided images. Inorder to properly view the stereoscopic images (one for each eye), theuser may wear the glasses 140 (although using some displays, glasses maynot be necessary). The glasses 140 may be anaglyph glasses, polarizedglasses, shuttering glasses, lenticular glasses, etc. Using anaglyphglasses, images for a first eye are presented according to a first color(and the corresponding lens has a corresponding color filter) and imagesfor a second eye are projected according to a second color (and thecorresponding lens has a corresponding color filter). With polarizedglasses, images are presented for each eye using orthogonalpolarizations, and each lens has the corresponding orthogonalpolarization for receiving the corresponding image. With shutteringglasses, each lens is synchronized to alternations of left and right eyeimages provided by the display(s) 150. The display may provide bothpolarizations simultaneously or in an alternating manner (e.g.,sequentially), as desired. Thus, the left eye is allowed to only seeleft eye images during the left eye image display time and the right eyeis allowed to only see right eye images during the right eye imagedisplay time. With lenticular glasses, images form on cyclindrical lenselements or a two dimensional array of lens elements. The stereoscopicimage may be provided via optical methods, where left and right eyeimages are provided only to the corresponding eyes using optical meanssuch as prisms, mirror, lens, and the like. Large convex or concavelenses can also be used to receive two separately projected images tothe user.

In one embodiment, the glasses 140 may be used as a position inputdevice to track the eyepoint of a user viewing a 3D scene presented bythe system 100. For example, the glasses 140 may provide informationthat is usable to determine the position of the eyepoint(s) of the user,e.g., via triangulation. The position input device can include aninfrared detection system to detect the position the viewer's head toallow the viewer freedom of head movement or use a light sensitivedetection system. Other embodiments of the input device can be thetriangulation method of detecting the viewer eyepoint location, such asa camera (e.g., a CCD camera) providing position data suitable for thehead tracking objectives of the invention. The input device can bemanually operated by the viewer, such as a keyboard, mouse, trackball,joystick, or the like, to indicate the correct display of the horizontalperspective display images. However, any method for tracking theposition of the user's head or eyepoint is envisioned. Accordingly, the3D scene may be rendered such that user can view the 3D scene withappropriately modified projection skewing (e.g., since it is based onthe eyepoint of the user). Thus, the 3D scene may be particularlyrendered for the eyepoint of the user, using the position input device.In some embodiments, each eyepoint may be determined separately, or asingle eyepoint may be determined and an offset may be used to determinethe other eyepoint.

The relationship among the position/orientation of the display(s) 150and the eye(s) position of the user may be used to map a portion of thevirtual space to the physical space of the system 100. In essence, thephysical space and components used are to be mapped to the virtual modelin order to accurately render a 3D scene of the virtual space. Examplesfor implementing such a system are described in theincorporated-by-reference U.S. patent application Ser. No. 11/098,681entitled “Horizontal Perspective Display” (U.S. Patent Publication No.US 2005/0219694).

One or more of the user input devices (e.g., the keyboard 120, the mouse125, the stylus 130, etc.) may be used to interact with the presented 3Dscene. For example, the user input device 130 (shown as a stylus), e.g.,a 3D pointing device, or simply the user's hands may be used to interactwith virtual objects of the 3D scene (via the viewed projected objects).However, this “direct” interaction may lend itself more easily to “openspace” portions of the 3D scene. Thus, at least a portion of the 3Dscene may be presented in this “open space”, which is in front of orotherwise outside of (e.g., behind) the at least one display. Thus, atleast a portion of the 3D scene may appear as a hologram above thesurface of the display 150. For example, when the horizontal display150B is used, the 3D scene may be seen as hovering above the horizontaldisplay, as illustrated by the young polar bear image of FIG. 1. Itshould be noted however, that a portion of the 3D scene may also bepresented as appearing below the display surface, which is not in “openspace”. Thus, “open space” refers to a space which the user is able tofreely move and interact with (e.g., where the user is able to place hishands in the space) rather than a space the user cannot freely move andinteract with (e.g., where the user is not able to place his hands inthe space, such as below the display surface). Note that it is possibleto have open space behind the display surface, e.g., where the user isable to put his hands behind the display surface and freely move around.Such embodiments may be particularly applicable for see-throughdisplays. This “open space” may be referred to as a “hands-on volume” asopposed to an “inner volume” or “inner space”, which may be under thesurface of the display(s). Thus, the user can interact with virtualobjects in the open space because they are proximate to the user's ownphysical space. The inner volume is located behind the viewing surfaceand presented objects appear inside the physically viewing device. Thus,objects of the 3D scene presented within the inner volume do not sharethe same physical space with the user and the objects therefore cannotbe directly, physically manipulated by hands or hand-held tools. Thatis, they may be manipulated indirectly, e.g., via a computer mouse or ajoystick, or via a 3D pointing device operating in an appropriate mode.

In some embodiments, this open space interaction may be achieved byhaving a 1:1 correspondence between the virtual objects (e.g., in thevirtual space) and projected objects (e.g., in the physical space).Thus, an accurate and tangible physical interaction is provided byallowing a user to touch and manipulate projected objects with his handsor hand held tools, such as the stylus 130. This 1:1 correspondence ofthe virtual elements and their physical real-world equivalents isdescribed in more detail in U.S. Patent Publication No. 2005/0264858,which was incorporated by reference in its entirety above. This 1:1correspondence is a new computing concept that may allow the user todirectly access and interact with projected objects of the 3D scene.This new concept requires the creation of a common physical referenceplane, as well as the formula for deriving its unique x, y, z spatialcoordinates, thereby correlating the physical coordinate environment tothe virtual coordinate environment. Additionally, the 1:1 correspondenceallows the user's movement of virtual objects or other interaction(e.g., via the stylus 130) to be the same in physical space and inpresented space. However, other embodiments are envisioned where thereis a ratio between the distance of the user's physical movement and thecorresponding movement in the presented 3D scene (e.g., of the presentedobject or virtual stylus).

As described below, the user may be able to interact with the 3D sceneusing various user interface (UI) elements, which may be displayedwithin the 3D scene, or in a 2D presentation that is not part of the 3Dscene.

The 3D scene generator stored and executed in the chassis 110 may beconfigured to dynamically change the displayed images provided by thedisplay(s) 150. More particularly, the 3D scene generator may update thedisplayed 3D scene based on changes in the user's eyepoint,manipulations via the user input devices, etc. Such changes may beperformed dynamically, at run-time. The 3D scene generator may also keeptrack of peripheral devices (e.g., the stylus 130 or the glasses 140) toensure synchronization between the peripheral device and the displayedimage. The system can further include a calibration unit to ensure theproper mapping of the peripheral device to the display images and propermapping between the projected images and the virtual images stored inthe memory of the chassis 110.

In further embodiments, the system 100 (e.g., the display(s) 150) canfurther comprise an image enlargement/reduction input device, an imagerotation input device, and/or an image movement device to allow theviewer to adjust the view of the projection images.

Thus, the system 100 may present a 3D scene which the user can interactwith (e.g., using UI elements or tools) in real time. The system maycomprise real time electronic display(s) 150 that can present or conveyperspective images in the open space and a peripheral device 130 thatmay allow the user to interact with the 3D scene with hand controlled orhand-held tools, e.g., 2D or 3D pointing devices. The system 100 mayalso include means to manipulate the displayed image such asmagnification, zoom, rotation, movement, and even display a new image.

Further, while the system 100 is shown as including horizontal display150B since it simulates the user's visual experience with the horizontalground, any viewing surface could offer similar 3D illusion experience.For example, the 3D scene can appear to be hanging from a ceiling byprojecting the horizontal perspective images onto a ceiling surface, orappear to be floating from a wall by projecting horizontal perspectiveimages onto a vertical wall surface. Moreover, any variation in displayorientation and perspective (or any other configuration of the system100) are contemplated.

FIG. 2 illustrates another embodiment of the system 100, shown as system200. In this embodiment, the system includes similar components as thesystem of FIG. 1, but in this exemplary embodiment utilizes a single 3Ddisplay. Below are described various novel techniques and mechanisms foruser interaction with such 3D display systems.

Exemplary Applications

Embodiments of the present invention may augment the current state ofreal-time computer-generated 3D computer graphics and tactilecomputer-human interfaces with real time interaction. More specifically,these new embodiments may enable real-time computer-generated 3Dsimulations to coexist in physical space and time with the userinteracting with the projected objects. This unique ability may beuseful in many industries including, but not limited to, electronics,computers, biometrics, medical, education, games, movies, science,legal, financial, communication, law enforcement, national security,military, print media, television, advertising, trade show, datavisualization, computer-generated reality, animation, CAD/CAE/CAM,productivity software, operating systems, and more.

FIG. 3A—Method of Invoking 3D Pointing Device Operational Modes Based onProximity to Display

FIG. 3A is a flowchart of an exemplary method for invoking 2D and 3Doperational modes of a 3D pointing device in a 3D presentation systembased on proximity to a display, e.g., a screen of a display device,according to one embodiment. The method shown in FIG. 3A may be used inconjunction with any of the systems or devices described herein. Invarious embodiments, some of the method elements shown may be performedconcurrently, in a different order than shown, or may be omitted.Additional method elements may also be performed as desired. As shown,the method may operate as follows.

In 302, a 3-dimensional (3D) stereoscopic scene may be displayed via atleast one stereoscopic display device. A simple exemplary 3Dstereoscopic scene is shown in FIG. 4A (as well as FIGS. 1, 2, andothers), specifically, a small house displayed on a horizontal displaydevice 150B. In other words, the 3D stereoscopic scene may be displayedin a stereoscopic manner on the at least one stereoscopic displaydevice. Note that this is not equivalent to displaying a 3D scene, e.g.,a computer generated scene using 3D modeling, in 2D on a 3D televisionoperating in 2D mode.

In 304, a 2-dimensional (2D) scene may be displayed via the at least onestereoscopic display device concurrent with display of the 3Dstereoscopic scene, where the entire 2D scene is displayed at zeroparallax, i.e., in the plane of the display device's screen. The2-dimensional (2D) scene may be on a single plane and that plane may beat zero parallax. In one exemplary embodiment, the 2D scene may be agraphical user interface (GUI) that includes one or more GUI elements,as discussed in detail below with reference to the method of FIG. 3B. Inanother exemplary embodiment, the 2D scene may be a background orbackdrop for the 3D stereoscopic scene with one or more 2D objects. Inother embodiments, the 2D scene may be any type of 2D scene as desired,including combinations of the above. Note that the 2D scene is notnecessarily in or of the (or a) 3D stereoscopic scene. For example, inan embodiment where the 2D (stereoscopic) scene is the operating system(OS) GUI, e.g., “desktop”, the OS desktop is not within a 3Dstereoscopic scene. If simultaneously, there is a stereo 3D application(presenting a 3D stereoscopic scene) taking up part of the desktop, thenboth the stereo 3D scene (from the application) and the normal 2D OSdesktop (2D scene) are displayed via the stereoscopic display device(s).

In 306, a current position of a cursor may be determined with respect tothe at least one stereoscopic display device based on a position of a 6degree of freedom (6DOF) 3D pointing device. In other words, the cursorposition may be based on the position of a 6DOF 3D pointing device inrelation to the at least one stereoscopic display device (e.g., screen),such as the (physical) stylus 130 shown in FIGS. 1 and 2. For example,the cursor may be an associated 3D cursor corresponding to the tip ofthe physical stylus 130. In an alternate embodiment or configuration,the cursor may be positioned at or may correspond to a point offset fromthe tip. In one embodiment, in being based on the position of the 6DOF3D pointing device, the current position of the cursor may be based on avirtual beam extending from the position of the 6DOF 3D pointing device.Moreover, the virtual beam may be a configurable length, e.g., theoffset from the 6DOF 3D pointing device to the cursor may be adjustable.The distal end of the virtual beam (i.e., the end of the beam furthestfrom the 6DOF 3D pointing device) may be or include a virtual stylustip, and the cursor may correspond to the virtual stylus tip.

In 308, the cursor may be displayed via the at least one stereoscopicdisplay device concurrent with display of the 3D stereoscopic scene andthe 2D scene. The cursor may operate in a 2D mode in response to thecurrent position being inside a specified volume proximate to the atleast one stereoscopic display device. Moreover, in the 2D mode, thecursor may be usable to interact with the 2D scene, which may be at zeroparallax. Similarly, the cursor may operate in a 3D mode in response tothe current position being outside the specified volume proximate to theat least one stereoscopic display device, where in the 3D mode, thecursor may be usable to interact with the 3D stereoscopic scene ineither zero, positive, or negative parallax. In some embodiments, in the2D mode the cursor may be displayed in the 2D scene with a firstrepresentation, and in the 3D mode the cursor may be displayed in the 3Dstereoscopic scene with a second representation, where the secondrepresentation is different from the first representation.

In some embodiments, the 2D scene may be or include a graphical userinterface (GUI) that includes one or more 2D user interface (UI)elements. In being usable to interact with the 2D scene, the cursor maybe usable to interact with the one or more 2D UI elements. Thus, in anembodiment where the 2D scene is or includes a GUI with one or more 2DGUI elements, in response to the current position being inside thespecified volume proximate to the at least one stereoscopic displaydevice, input regarding a displayed 2D (graphical) user interface (UI)element may be received. In other words, if the current position iswithin a specified volume of space, e.g., adjacent to the display (e.g.,screen), pointing device operations (with or of the 6DOF 3D pointingdevice) may pertain to a 2D GUI element, such as a menu. In oneexemplary embodiment, illustrated in FIG. 4A, the volume may bespecified or bounded by a plane 450 parallel to the display (screen) andthe screen, such that if the current position is between the plane andthe display, input from the 6DOF 3D pointing device, e.g., stylus 130(or other pointing device), may be directed to the 2D UI element, e.g.,menu item 406, i.e., the specified volume may be defined by a planeparallel to a screen of the at least one stereoscopic display device,where the specified volume is on the screen side of the plane.

As FIG. 4A shows, in this exemplary embodiment the current position isbelow such a plane, and thus, is within the specified volume proximate(i.e., adjacent or near) to the display (screen). In some embodiments,when the current position is in this specified volume, the cursor maychange its appearance to reflect a 2D operational mode, as FIG. 4Aindicates. Moreover, in one embodiment, when in the 2D mode, the cursormay “pop” to the 2D surface of the display (zero parallax), and thus mayoperate like a standard 2D mouse cursor, and may facilitate userinteractions with 2D GUI elements or 2D objects displayed in the 2Dscene. The cursor itself may change appearance to resemble one of many2D cursor appearances. Said another way, when in 2D mode, the cursor maybe displayed as a 2D element on the display, e.g., 2D cursor 402, ratherthan a 3D cursor in the space in front of the surface of the display (or“above”, i.e., negative parallax), or behind the surface of the display(or “into” or “below”, i.e., positive parallax). Conversely, when thecurrent position of cursor (whose position may be computed, but notdisplayed, since the cursor is in 2D mode, and thus displayed on thesurface of the display) leaves the specified volume, the cursor mayrevert to its 3D appearance and position.

In some embodiments, the specified volume may be defined by respectivebounding volumes proximate to each of one or more 2D user interface (UI)elements on the at least one stereoscopic display device. Alternatively,in some embodiments, the specified volume may be defined by exclusion ofa respective bounding volume of at least one object in the 3Dstereoscopic scene. In other words, the specified volume demarcating the2D and 3D modes of operation may be the volume or space outside thevolume or space around one or more 3D objects in the 3D stereoscopicscene, e.g., outside convex hulls associated with each 3D object.

In some embodiments, the 2D UI element(s) may not be displayed unlessthe 3D pointing device is operating in 2D mode, i.e., the 2D UI may onlybe displayed when the user positions the 6DOF 3D pointing device orassociated cursor within the specified volume. In yet anotherembodiment, the 2D UI element(s) may displayed in a partial orintermediate manner until the 6DOF 3D pointing device or cursor isoperating in 2D mode, i.e., the 2D UI may only be fully displayed whenthe user positions the 6DOF 3D pointing device or associated cursorwithin the specified volume. For example, the 2D UI element(s) may besemi-transparent, or “grayed out” until the 2D mode is active. Ofcourse, in various embodiments, any other type of partial orintermediate display techniques may be used as desired.

In one embodiment, rather than distance from the stereoscopic displaydevice, the orientation of the 6DOF 3D pointing device or cursor, e.g.,the angle of the positioning of the stylus and/or its cursor beam, maydictate or invoke the change to 2D menu mode. Of course, in otherembodiments, any degrees of freedom may be used as desired.

Similarly, as discussed above, the cursor may operate in 3D mode inresponse to the current position being outside the specified volume, andso in some embodiments, input regarding the 3D scene may be received viathe 6DOF 3D pointing device or associated cursor. In some embodiments,the 3D stereoscopic scene may include at least one 3D graphical object,and in being usable to interact with the 3D stereoscopic scene, thecursor may be usable to interact with the at least one 3D graphicalobject. In other words, if the current position is not in the specifiedvolume (e.g., proximate to the display), the 3D pointing device (e.g.,stylus) may operate in a 3D mode, and thus may be used to interact withobjects in the 3D scene, e.g., the house.

FIG. 4B illustrates this feature following the example of FIG. 4A. Asmay be seen, in this exemplary embodiment, the current position (of thestylus 130 or its associated 3D cursor) is above the plane 450 and thusmay operate in 3D mode, e.g., allowing user interaction with respect tothe house. Note that in this embodiment, the cursor has reverted to its3D representation, e.g., 3D cursor 404.

As discussed above, in some embodiments, the (3D) cursor may bepositioned at or near the tip of the 6DOF 3D pointing device, asillustrated in FIG. 4B. However, in some embodiments, the 3D pointingdevice may be capable of extending the 3D cursor in a specifieddirection, e.g., in the form of a ray or beam of configurable length.FIG. 4C illustrates an exemplary embodiment where the cursor is shownextended from the tip of the stylus 130, e.g., 3D extended cursor 408.Thus, it may be the case that the stylus tip (e.g., position of the 3Dpointing device) may be outside the specified volume (e.g., above theplane) while the 3D cursor may be positioned within the specified volume(e.g., below the plane). Accordingly, in some embodiments, theparticular position used may be configurable, or may default to either.For example, in one embodiment, the current position may always be thatof the 3D cursor associated with the 3D pointing device.

It should be noted that in some embodiments, the specified volume mayhave different shapes or arrangements. For example, as noted above, insome embodiments, the volume may be defined by one or more boundingvolumes, e.g., rectangular solids, spheres, convex hulls, etc.,containing respective objects of the 3D scene, where the specifiedvolume proximate to the display may be the space excluded by thesevolumes. In other words, the specified volume proximate to the displaymay be defined by not being a volume specified for 3D mode. Said anotherway, in some embodiments, the “2D mode” volume may be defined bydefining the “3D mode” volume, where the “2D mode” volume does notinclude the “3D mode” volume. In other embodiments, the specified volumemay include respective volumes defined around (or above) 2D UI elements,such that if the current position is within any of these volumes, the 2Dmode is operational. In one embodiment, the specified volume and thespace outside the specified volume may define respective regions, whereone region is displayed as inactive when the cursor is in the otherregion.

In some embodiments, the method may repeat the above method elements oneor more times in an iterative manner.

Thus, in various embodiments, the cursor (and/or the 6DOF 3D pointingdevice), or more generally, the 3D presentation system, may operate indifferent modes, e.g., 2D vs. 3D, based on proximity of the cursor tothe at least one stereoscopic display device, or more generally, basedon positioning the 3D pointing device or its associated 3D cursor withina specified volume (or alternatively, outside another specified volume).

FIG. 3B—Method of Invoking 3D Pointing Device Operational Modes Based onProximity to Display

FIG. 3B is a flowchart of an exemplary method for invoking 2D and 3Doperational modes of a 3D pointing device in a 3D presentation systembased on proximity to a display, e.g., a screen of a display device,according to further embodiments. More specifically, the method of FIG.3B is a variant of the more general method of FIG. 3A. The method shownin FIG. 3B may be used in conjunction with any of the systems or devicesdescribed herein. In various embodiments, some of the method elementsshown may be performed concurrently, in a different order than shown, ormay be omitted. Additional method elements may also be performed asdesired. For the reader's convenience, some of the features describedabove with respect to embodiments of the method of FIG. 3A that alsoapply to embodiments of the method of FIG. 3B are re-presented in thebelow description of FIG. 3B. As shown, the method may operate asfollows.

In 312, a 3-dimensional (3D) scene may be presented via at least onedisplay, including displaying at least one stereoscopic image. A simpleexemplary 3D scene is shown in FIG. 4A (as well FIGS. 1, 2, and others),specifically, a small house displayed on a horizontal display device150B.

In 314, a current position of a 3D pointing device or an associated 3Dcursor with respect to the at least one display may be determined. Inother words, the method may determine the current position of a 3Dpointing device, such as the stylus 130 shown, or its associated 3Dcursor, in relation to the display.

In 316, in response to the current position being inside a specifiedvolume proximate to the at least one display, input regarding a 2D(graphical) user interface (UI) element displayed on the at least onedisplay may be received. In other words, if the current position iswithin a specified volume of space, e.g., adjacent to the display (e.g.,screen), pointing device operations may pertain to a 2D GUI element,such as a menu. In one exemplary embodiment, illustrated in FIG. 4A, thevolume may be specified or bounded by a plane 450 parallel to thedisplay (screen) and the screen, such that if the current position isbetween the plane and the display, input from the stylus (or otherpointing device) may be directed to the 2D UI element, e.g., menu item406. In other words, as FIG. 4A shows, in this exemplary embodiment thecurrent position is below such a plane, and thus, is within thespecified volume proximate (i.e., adjacent or near) to the display. Insome embodiments, when the current position is in this specified volume,the cursor may change its appearance to reflect a 2D operational mode,as FIG. 4A indicates. Moreover, in one embodiment, when in this 2D mode,the cursor may “pop” to the 2D surface of the display, and thus mayoperate like a standard 2D mouse cursor. Said another way, when in 2Dmode, the cursor may be displayed as a 2D element on the display, e.g.,2D cursor 402, rather than a 3D cursor in the space above (or below) thesurface of the display. Conversely, when the current position of the 3Dpointing device or the associated 3D cursor (whose position may becomputed, but not displayed, since the cursor is in 2D mode, and thusdisplayed on the surface of the display) leaves the specified volume,the cursor may revert to its 3D appearance and position.

In some embodiments, the 2D UI element(s) may not be displayed unlessthe 3D pointing device is operating in “2D mode”, i.e., the 2D UI mayonly be displayed when the user positions the 3D pointing device orassociated cursor within the specified volume. In yet anotherembodiment, the 2D UI element(s) may displayed in a partial orintermediate manner until the 3D pointing device is operating in “2Dmode”, i.e., the 2D UI may only be fully displayed when the userpositions the 3D pointing device or associated cursor within thespecified volume. For example, the 2D UI element(s) may besemi-transparent, or “grayed out” until the 2D mode is active. Ofcourse, in various embodiments, any other type of partial orintermediate display techniques as desired.

In one embodiment, rather than distance from the display, theorientation of the 3D pointing device or cursor, e.g., the angle of thepositioning of the stylus and/or its cursor beam, may dictate or invokethe change to 2D menu mode. Of course, in other embodiments, any degreesof freedom may be used as desired.

In 318, in response to the current position being outside the specifiedvolume proximate to the at least one display, input regarding the 3Dscene may be received. In other words, if the current position is not inthe specified volume proximate to the display, the 3D pointing device(e.g., stylus) may operate in a 3D mode, and thus may be used tointeract with objects in the 3D scene, e.g., the house.

FIG. 4B illustrates this feature following the example of FIG. 4A. Asmay be seen, in this exemplary embodiment, the current position (of thestylus 130 or its associated 3D cursor) is above the plane 450 and thusmay operate in 3D mode, e.g., allowing user interaction with respect tothe house. Note that in this embodiment, the cursor has reverted to its3D representation, e.g., 3D cursor 404.

In some embodiments, the 3D cursor may be positioned at or near the tipof the 3D pointing device, as illustrated in FIG. 4B. However, in someembodiments, the 3D pointing device may be capable of extending the 3Dcursor in a specified direction, e.g., in the form of a ray or beam ofconfigurable length. FIG. 4C illustrates an exemplary embodiment wherethe cursor is shown extended from the tip of the stylus 130, e.g., 3Dextended cursor 408. Thus, it may be the case that the stylus tip (e.g.,position of the 3D pointing device) may be outside the specified volume(e.g., above the plane) while the 3D cursor may be positioned within thespecified volume (e.g., below the plane). Accordingly, in someembodiments, the particular position used may be configurable, or maydefault to either. For example, in one embodiment, the current positionmay always be that of the 3D cursor associated with the 3D pointingdevice.

It should be noted that in some embodiments, the specified volume mayhave different shapes or arrangements. For example, in some embodiments,the volume may be defined by one or more bounding volumes, e.g.,rectangular solids, spheres, convex hulls, etc., containing respectiveobjects of the 3D scene, where the specified volume proximate to thedisplay may be the space excluded by these volumes. In other words, thespecified volume proximate to the display may be defined by not being avolume specified for 3D mode. Said another way, in some embodiments, the“2D mode” volume may be defined by defining the “3D mode” volume, wherethe “2D mode” volume does not include the “3D mode” volume. In otherembodiments, the specified volume may include respective volumes definedaround (or above) 2D UI elements, such that if the current position iswithin any of these volumes, the 2D mode is operational.

In some embodiments, the method may repeat the above method elements oneor more times in an iterative manner.

Thus, in various embodiments, the 3D pointing device, or more generally,the 3D presentation system, may operate in different modes, e.g., 2D vs.3D, based on proximity of the 3D pointing device or its associated 3Dcursor to the display, or more generally, based on positioning the 3Dpointing device or its associated 3D cursor within a specified volume(or alternatively, outside another specified volume).

It should be noted that any of the features and details described abovewith respect to the method of FIG. 3A may also be incorporated inembodiments of the method of FIG. 3B.

Further Exemplary Embodiments

The following describes further exemplary embodiments.

In one embodiment, a 3-dimensional (3D) scene may be presented by atleast one display. Presenting the 3D scene may include displaying atleast one stereoscopic image of the 3D scene by the at least onedisplay, where the 3D scene is presented according to a first viewpointwith respect to the at least one display, and where the 3D scenecorrelates to a physical open space. The first viewpoint may have afirst viewpoint X, Y, and Z location and a first pitch, yaw and rollorientation when referenced to the at least one display.

The 3D scene may include a rendering of at least one object at leastpartially visible from the first viewpoint within the 3D scene, and the3D scene may include at least one 3D object rendered in stereo 3D.

A 2D scene may be presented by the at least one display. Presenting the2D scene may include displaying at least one stereoscopic image of the2D scene by the at least one display. The 2D scene and the 3D scene maybe concurrently presented within the at least one display.

A first virtual viewpoint within the 3D scene may be determined, wherethe first virtual viewpoint is different than the first viewpoint. Thefirst virtual view point may correspond to a first X, Y, and Z locationand a first pitch, yaw and roll orientation in physical open space andmay map to a first coordinate in the 3D scene, where the firstcoordinate includes a second X, Y, and Z location in the 3D scene, andwhere the first coordinate further includes a second pitch, yaw and rollorientation in the 3D scene.

A first region and a second region may be determined, where the firstregion correlates to the 3D scene, and where the second region isoutside the 3D scene and includes the 2D scene.

A cursor may be presented in a 2D operational mode when the determinedfirst virtual viewpoint is determined to be in the second region, andthe cursor may be presented in a 3D appearance and position when thedetermined first virtual viewpoint is determined to be in the firstregion.

It should be noted that any of the features and details described abovewith respect to the methods of FIGS. 3A and 3B may also be incorporatedin embodiments of the above method. More generally, any of the variousfeatures and limitations disclosed herein may be used in any combinationdesired.

In another implementation, the change in the user's viewpoint may alterthe user's perspective and hence the system's rendering of the regions,independent of the positioning of the cursor. The cursor in thisimplementation may then be located in an alternate region and mayoperate or perform in accordance with the region that it is now within.

User Interactions in a 3D Presentation System

In some 3D presentation systems, a user may interact with the 3D scene,e.g., with objects in the 3D scene, e.g., identifying or selecting anobject, and applying some operation to the object via a 3D pointingdevice, e.g., moving, rotating, or otherwise manipulating the object, orperforming some other operation with respect to the device that may notchange the object, e.g., moving a virtual cameral or viewpoint in anorbit around some identified object while keeping the object in thecenter of the view.

When performing such an operation, e.g., rotating an object orperforming any effect on an object (e.g., coloring, scaling, etc.) thereis usually a control mechanism provided, e.g., a slider or a mousegesture, to modulate the effect. In other words, for some effects oroperations, the user may configure the degree to which, or rate atwhich, the effect is applied or manifested. Said another way, the usermay be allowed to configure the rate of the effect, much as a user mayconfigure the speed a cursor moves in response to a given movement of amouse.

However, current techniques are limited to: using 2D mouse positioningfor effecting speed of the effect, only one effect being controllablevia the 2D mouse positioning, or a two step approach being requiredconfigure or control the rate of effect (e.g., adjust slider for speedsetting, then use the mouse to control the effect at the speed set bythe slider). The below describes a novel way to dynamically controleffect rates via a 3D pointing device which has more than three degreesof freedom, according to various embodiments.

FIG. 5—Flowchart of a Method for User Interactions in a 3D PresentationSystem Using a 3D Pointing device

FIG. 5 is flowchart of a method for user interactions in a 3Dpresentation system using a 3D pointing device, according to someembodiments. The method shown in FIG. 5 may be used in conjunction withany of the systems or devices described herein. In various embodiments,some of the method elements shown may be performed concurrently, in adifferent order than shown, or may be omitted. Additional methodelements may also be performed as desired. As shown, the method mayoperate as follows.

In 502, a 3-dimensional (3D) scene that includes at least one object maybe presented via at least one display, including displaying at least onestereoscopic image. The at least one object may be identified. Note thatas used herein, the term “identified” means that the object has beendistinguished, e.g., from other (unidentified or de-identified) objects.In some embodiments, being “identified” may indicate “pre-selection” ofan object, where, for example, a 3D pointing device may be pointing atthe object, e.g., positioning an associated 3D cursor on (or in) theobject, where, for example, the object may be highlighted, but may notbe selected (which may require clicking on the object, or releasing abutton to complete such a click). In other embodiments, being“identified” may mean being “selected”.

A simple exemplary 3D scene with at least one object is shown in FIG.6A, specifically, a human head is shown displayed on (or above, e.g.,via stereoscopic techniques) a horizontal display device 150B. The atleast one object may be identified in response to an action performedwith the 3D pointing device, being pointed to, clicked, or half-clicked(button press, no release), or selected.

In 504, a change in geometric state of the 3D pointing device or anassociated 3D cursor may be determined. The 3D pointing device may haveat least 4 degrees of freedom (DOF), including at least one DOF thatcorresponds to an operation with respect to the at least one object, andat least one other DOF that corresponds to a rate of the operation. Thechange in geometric state may specify the operation and the rate of theoperation. FIG. 6A illustrates a first geometric state (of the 3Dpointing device or associated 3D cursor), and FIGS. 6B and 6C illustraterespective second geometric states, described below in more detail.

In some embodiments, the at least 4 degrees of freedom may correspond toa reference frame, and the change in geometric state may be or include achange in the reference frame. For example, in one embodiment, the atleast 4 degrees of freedom may include position with respect to one ormore spatial axes, and/or orientation with respect to the one or morespatial axes. Accordingly, the change in geometric state of the 3Dpointing device or associated 3D cursor may include a change in positionwith respect to the one or more spatial axes and/or orientation withrespect to the one or more spatial axes. In some embodiments, the atleast 4 degrees of freedom may be or include 5 or 6 degrees of freedom.For example, the degrees of freedom may include 3 positional degrees offreedom, and 3 orientation degrees of freedom.

In 506, the specified operation may be performed with respect to the atleast one object at the specified rate. In other words, the geometricstate change with respect to the at least one DOF that corresponds tothe operation may invoke the operation, and the geometric state changewith respect to the at least one other DOF that corresponds to the rateof the operation may determine the rate at which the operation isperformed.

Any of a variety of operations may be applied per the above, includingfor example, but not limited to, one or more of: rotation of the atleast one object, movement (i.e., translation) of the at least oneobject, movement of a point of view (POV) with respect to the at leastone object, e.g., movement of a virtual camera, scaling of the at leastone object, color of the at least one object, brightness of the at leastone object, or changing shape of the at least one object, among others.

In the exemplary embodiment of FIGS. 6A-6C, a rotation operation isperformed in response to movement of the 3D cursor parallel to thedisplay (screen) at a rate specified by the distance of the cursor fromthe display (screen). More specifically, in FIGS. 6A and 6B, the cursorthe cursor is at a distance H1, as shown, and in FIG. 6B, the cursor hasbeen moved (horizontally, to the right) a distance M, resulting in a 45degree rotation around the vertical axis. Now, in FIG. 6C, the cursor isat a distance H2, with H2 greater than H1, and the resulting rotation ofthe object (head) based on the same movement M is 90 degrees. Of course,the operation and rates shown are exemplary only, and any otheroperations and rates may be utilized as desired.

Note that a corresponding movement, also horizontal (or more generally,parallel to the display), but perpendicular to the vertical axis, mayresult in a rotation around a different axis, e.g., a horizontal axis(in this horizontal display embodiment).

In some embodiments, at least two of the degrees of freedom may togethercorrespond to the operation, and one other DOF may correspond to therate of the operation. For example, in a rotation embodiment, themovement of the cursor may be oblique, e.g., with components along twodifferent axes, e.g., x and y, and at a determined distance from thescreen (e.g., normal to the screen, along the z axis) and the resultingrotation operation may thus be around an axis perpendicular to themovement, with the rotation amount for a given oblique movement of thecursor being specified by the z distance. In other words, the resultingrotation may be a combination of Euler angles.

In another embodiment, at least two of the degrees of freedom maytogether correspond respectively to components of the operation, but twoother degrees of freedom may correspond respectively to components ofthe rate of the operation. In other words, in addition to operationcomponents (e.g., rotation components, as described above), there may berate components that may differ from one another. Thus, a given zdistance may specify different rates for the different components of therotation.

In a further embodiment, an orientation of the 3D point device orassociated cursor, e.g., the angle of a stylus, may either effect therate, or may determine scaling (e.g., based on the increments of say 10degrees in one direction or another).

In some embodiment, the change in geometric state may be quantized byconfigurable increments. For example, the rate for an operation maychange discretely and by a specified amount per each (specified)increment of the movement per the operation DOF(s). Moreover, in someembodiments, the 3D pointing device may be configurable, e.g., via abutton, to change the increment, thereby setting the operation rate,e.g., scaling, at faster or slower rates.

It should be noted that in various embodiments, any of the techniquesand mechanisms disclosed herein may be used in conjunction as desired.For example, in one exemplary embodiment that combines aspects of themethod of FIG. 3A or 3B with that of FIG. 7, the 3D cursor (or pointingdevice's) rate specification via H (e.g., normal displacement fromdisplay screen) may not go into effect unless the position is outsidethe specified volume that specifies the 2D mode (for 2D GUI operations).Thus, if the plane defining the volume is, say, 10 mm (or 10 cm, etc.),then the rate determination may only occur when the cursor's distancefrom the display exceeds this value.

In some embodiments, one or more of the degrees of freedom may beconfigurable to be locked. Accordingly, the change in geometric state ofthe 3D pointing device or associated 3D cursor with respect to thelocked one or more degrees of freedom may not change the operation orthe rate of the operation.

In one embodiment, the change in geometric state may be a movement (orother change) with respect to an anchor point in the 3D scene or on thedisplay. Thus, the change in geometric state may be with respect to thispoint. In various embodiments, the anchor point may be or include one ormore of: a default position, a position of the at least one object, or auser-specified position, among others, as desired.

For example in one exemplary embodiment, the anchor point may be thecenter of the screen (or screen plane) by default. As another example,the anchor point may be the position of the identified or selectedobject. In another exemplary embodiment, the anchor point may bedetermined by the user by clicking on a position, e.g., in the 3D scene,where a subsequent movement invoking the operation may be with respectto this point.

In some embodiments, the anchor may be a reference frame, and mayspecify a value with respect to more than just spatial DOFs, e.g., mayalso be with respect to orientation with respect to one or more axes.

Identification of Objects in a 3D Scene

In many 3D presentation applications, there is a need to identify orselect objects that are somehow related to an already identified object.For example, the identified object may be in an object hierarchy orsequence, and other objects in the hierarchy or sequence may need to beidentified. However, sometimes the hierarchy (or other relationships) ofobjects is difficult to see, and therefore it may be difficult toindicate the appropriate objects with a cursor, e.g., when there is theneed to select higher or lower level objects (in a hierarchy) from acurrent highlighted object, the user may be required to position thecursor exactly, e.g., extending the cursor to include the sought objects(objects of interest) within the hierarchical arrangement of objects. Inaddition prior art techniques that use a direct concatenated selectionprocess of successively picking more and more objects to the collectionmay be cumbersome, especially with structures or hierarchies that have alarge number of objects.

Regarding the various 3D techniques disclosed herein, it should be notedthat in some stereoscopic 3D systems, i.e., systems that utilize stereovisioning techniques to simulate 3D viewing, the 3D cursor may be movedto positions above and below (or in front of, and behind) the displayscreen surface. In other words, positions of the 3D cursor may not berestricted to the volume in front of and adjacent to the display screen.Thus, for example, in some stereo visioning systems, some objects in adisplayed 3D scene may be positioned in the plane of the screen, somemay be positioned above or in front of the screen plane, and others maybe positioned behind or below the screen plane. Accordingly, also usingsuch stereo visioning techniques, the 3D cursor may be positioned by auser with respect to any of these objects, or with respect to any 2D or3D point or reference frame, as desired.

FIG. 7—Flowchart of a Method for Identifying Objects in a 3D Scene via a3D Pointing Device

FIG. 7 is a flowchart diagram illustrating one embodiment of a methodfor identifying objects in a 3D scene, according to some embodiments.The method shown in FIG. 7 may be used in conjunction with any of thesystems or devices described herein. In various embodiments, some of themethod elements shown may be performed concurrently, in a differentorder than shown, or may be omitted. Additional method elements may alsobe performed as desired. As shown, the method may operate as follows.

In 702, a 3-dimensional (3D) scene that includes a plurality of objectsmay be presented via at least one display, including displaying at leastone stereoscopic image. The plurality of objects may have spatial orassociative relationships, and may include at least one identifiedobject. As noted above, the term “identified” means that the object hasbeen indicated or distinguished, e.g., from other (unidentified orde-identified) objects. As also mentioned above, in some embodiments,being “identified” may indicate “pre-selection” of an object, where, forexample, a 3D pointing device may be pointing at the object, e.g.,positioning an associated 3D cursor on (or in) the object, where, forexample, the object may be highlighted, but may not be selected (whichmay require clicking on the object, or releasing a button to completesuch a click); however, in further embodiments, being “identified” maymean being “selected”.

In 704, one or more additional objects of the plurality of objects maybe identified based on the spatial or associative relationships inresponse to a difference between a current reference frame of a 3Dpointing device or an associated 3D cursor and an anchor referenceframe, resulting in a plurality of identified objects. The referenceframe of the 3D pointing device or associated 3D cursor and the anchorreference frame may be independent of positions of the at least oneidentified object and the plurality of identified objects.

FIGS. 8A-8C illustrate exemplary identification of objects in a 3D sceneper the method of FIG. 7, according to some embodiments. FIG. 8Aillustrates display of an exemplary 3D scene with multiple objectscomposing a human bust (head, neck, and shoulders), which form an objecthierarchy. As FIG. 8A shows, in this exemplary embodiment, the at leastone identified object is nose 802, pointed to by the 3D pointing device130, indicated by a dashed curve around the nose, and as furtherindicated in the object list shown above the head.

FIG. 8B illustrates identification of additional objects in thehierarchy one step or associative connection from the identified nose,specifically, a face object, in response to the 3D pointing device orcursor moving away from the nose (or display). Similarly, FIG. 8Cillustrates identification of further additional objects in thehierarchy two steps or associative connections from the identified nose,specifically, adding a scalp object to the identified face (includingnose) object, in response to the 3D pointing device or cursor movingaway from the nose (or bust, or display).

Describing the above exemplary embodiment in a slightly differentmanner, FIG. 8A shows that the nose was selected. Then from a particularposition, the user starts to move the cursor away from the nose. Thesystem knows the hierarchical structure of the objects, and furtherknows that the next higher elemental object in the hierarchy is theface. Accordingly, FIG. 8B illustrates the moved away position of thecursor and the corresponding highlighting (via dashed border) of theface (which is higher up the hierarchy from the nose) and hence thewhole of the face, including the eyes, cheek brow, forehead. Then as thecursor is pulled away further, as shown in FIG. 8C, the hierarchy ofobjects identified expands to the head, including face and scalp. Notethat if the cursor were pulled away even further (not shown), theidentified objects may include all of the body hierarchy or structure,e.g., including neck and chest.

Note that in other embodiments, this technique may be used with 2Dscenes and/or with a 2D pointing device, such as a conventional mouse.

In various applications, different scenarios or relationships may bedefined for objects within a space, in a hierarchy, or sequence, e.g., atemporal sequence (such as an animation sequence), or any combination.In other words, the spatial or associative relationships comprise aspatial arrangement of the plurality of objects, a hierarchy of theplurality of objects, a temporal sequence of the plurality of objects,and/or a network of objects.

Similar to the technique of the method of FIG. 5, in some embodiments,the reference frame or the anchor reference frame may correspond to atleast 4 degrees of freedom, and in some embodiments, may include 5 or 6DOFs, as desired (or even more, dependent on the pointing device anddisplay technology). Thus, for example, in some embodiments, thedifference between the current reference frame of the 3D pointing deviceor associated 3D cursor and the anchor reference frame may includeposition with respect to one or more spatial axes and/or orientationwith respect to the one or more spatial axes. The anchor point mayinclude one or more of: a default position or orientation, a position ororientation of the at least one object, or a user-specified position ororientation.

For example, in one exemplary embodiment directed to simple spatialarrangements of objects, e.g., where objects may be in proximity to eachother or even within each other, but not part of a defined hierarchy, auser may identify or select one object, then may place the cursor at astarting point and move away from the object, resulting inidentification (or selection) of other objects in proximity to theinitially identified (or selected) object, e.g., either one-to-onecorrespondence or in a scaled manner (e.g., per the techniques describedabove with respect to the method of FIG. 5). In other words, as the usermoves the cursor in a specified direction (or otherwise changes thereference frame of the pointing device with respect to one or morespecified DOFs), objects closest to the initially identified object maybe identified, then the next nearest, and so forth. Note that thedirection and or movement of the cursor may be independent of theplacement (positions) of the objects.

In one embodiment, the at least one identified object may be or includemultiple identified objects, and the method may de-select one or more ofthe multiple identified objects in response to a difference betweenanother current reference frame of the 3D pointing device or associated3D cursor and the anchor reference frame, resulting in de-identificationof at least one object of the multiple identified objects, where,similar to above, the other current reference frame of the 3D pointingdevice or associated 3D cursor and the anchor reference frame may beindependent of positions of the multiple identified objects. In otherwords, in one embodiment following the above example, if many objectsare already identified (or selected), then as the cursor is moved closerto the identified or selected grouping of objects (e.g., towards thecenter of the grouping) or towards a reference point or frame ofreference (e.g., towards an anchor point or reference frame), the outerobjects may becoming de-identified (or de-selected), followed by otherobjects closer to the center, as the movement continues. In other words,in one embodiment, movement in one direction may progressively identifyfurther objects (e.g., from the inside out), and movement in theopposite direction may de-identify (or de-select) objects (e.g., fromthe outside in).

In an exemplary embodiment directed to an object within a hierarchy ofobjects, the user may select one object within the hierarchy, then placethe cursor at a starting point and move away from the object, which mayresult in identification of objects higher up the hierarchy. Conversely,by moving the cursor closer to the selection of objects (or towards thecenter of the identified object or group of objects), successive outerlevel objects may be de-identified or de-selected, again, either with aone-to-one correspondence or in a scaled manner.

In an embodiment directed to timed events, e.g., a temporal sequence,such as in an animation, based on a selected object (e.g., frame) at aparticular time, as the cursor is moved away from the object, then oneor more time conditions (increasing or decreasing in time or bothsimultaneously may be selected and added (or subtracted) from theidentified (or selected) group. Of course, in other embodiments, any ofthe above embodiments may be used in any combinations desired.

As mentioned above, in some embodiments, the identification techniquemay be implemented using standard 2D tools (e.g., a mouse), where the 2Dplacement of the cursor may determine the hierarchical identificationand de-identification (or selection and de-selection). Similarly, instereoscopic 3-space (e.g., a 3D scene imaged in stereo), the proximityof the pointing device or associated cursor and its movement may effectthe hierarchy/spatial/temporal identification or de-identification (orselection or de-selection).

Also, there may be button controls to shift state of the selection. Forexample, the selection may only occur when a button is depressed as thecursor position is moved.

Thus, embodiments of the above technique may implement 2D or 3D cursorposition based identification or selection of objects in a spatial ortemporal hierarchy or other associative relationship, e.g., a network,where the position of the moving cursor is independent of the locationof the objects being selected (or de-selected) in relation to itsposition to the cursor.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

I claim:
 1. A non-transitory computer accessible memory medium thatstores program instructions executable by a processor to implement:display a 3-dimensional (3D) stereoscopic scene via at least onestereoscopic display device; determine a current position of a cursorwith respect to the at least one stereoscopic display device based on aposition of a 6 degree of freedom (6DOF) 3D pointing device with respectto the at least one stereoscopic display device; and display the cursorvia the at least one stereoscopic display device concurrent with displayof the 3D stereoscopic scene; wherein in response to the currentposition being inside a specified volume proximate to the at least onestereoscopic display device, the program instructions are furtherexecutable by the processor to implement display of a 2 dimensional (2D)scene via the at least one stereoscopic display device concurrent withdisplay of the 3D stereoscopic scene, wherein at least a portion of the2D scene overlaps at least a portion of the 3D stereoscopic scene,wherein the cursor operates in a 2D mode, wherein the entire 2D scene isdisplayed within a plane of a screen of the at least one stereoscopicdisplay device, wherein the specified volume is bounded by a planeparallel to a screen of the at least one stereoscopic display device,wherein the plane is a specified non-zero distance from the screen,wherein the specified volume is on the screen side of the plane, whereinboundaries of the specified volume are defined by boundaries of the atleast one stereoscopic display device, and wherein, in the 2D mode, thecursor is usable to interact with the 2D scene; and wherein in responseto the current position being outside the specified volume proximate tothe at least one stereoscopic display device, the program instructionsare further executable by the processor to implement removal of thedisplay of the 2D scene, wherein the cursor operates in a 3D mode,wherein, in the 3D mode, the cursor is usable to interact with the 3Dstereoscopic scene.
 2. The non-transitory computer accessible memorymedium of claim 1, wherein the specified volume defines a first regionand space outside the specified volume defines a second region, andwherein the second region is displayed as inactive when the cursor is inthe first region.
 3. The non-transitory computer accessible memorymedium of claim 1, wherein the 2D scene comprises a graphical userinterface (GUI) that includes one or more 2D user interface (UI)elements, and wherein, in being usable to interact with the 2D scene,the cursor is usable to interact with the one or more 2D UI elements atzero parallax.
 4. The non-transitory computer accessible memory mediumof claim 3, wherein the specified volume is further defined byrespective bounding volumes proximate to each of one or more 2D userinterface (UI) elements on the at least one stereoscopic display device.5. The non-transitory computer accessible memory medium of claim 1,wherein the specified volume is further defined by exclusion of arespective bounding volume of at least one object in the 3D stereoscopicscene.
 6. The non-transitory computer accessible memory medium of claim1, wherein the 3D stereoscopic scene includes at least one 3D graphicalobject; and wherein, in being usable to interact with the 3Dstereoscopic scene, the cursor is usable to interact with the at leastone 3D graphical object at zero, positive, or negative parallax.
 7. Thenon-transitory computer accessible memory medium of claim 1, wherein inbeing based on the position of the 6DOF 3D pointing device, the currentposition of the cursor is based on a virtual beam extending from theposition of the 6DOF 3D pointing device.
 8. The non-transitory computeraccessible memory medium of claim 7, wherein the virtual beam has aconfigurable length.
 9. The non-transitory computer accessible memorymedium of claim 7, wherein a distal end of the virtual beam comprises avirtual stylus tip, and wherein the cursor corresponds to the virtualstylus tip.
 10. The non-transitory computer accessible memory medium ofclaim 1, wherein the 6DOF 3D pointing device comprises a physicalstylus, and wherein the cursor corresponds to a tip of the physicalstylus.
 11. The non-transitory computer accessible memory medium ofclaim 1, wherein in the 2D mode the cursor is displayed in the 2D scenewith a first representation, and wherein in the 3D mode the cursor isdisplayed in the 3D stereoscopic scene with a second representation,wherein the second representation is different from the firstrepresentation.
 12. A system, comprising: a processor; at least onestereoscopic display device coupled to the processor; a 6 degree offreedom (6DOF) 3D pointing device coupled to the processor, andconfigured to specify a current position of a cursor with respect to theat least one stereoscopic display device based on a position of the 6DOF3D pointing device with respect to the at least one stereoscopic displaydevice; and a memory medium coupled to the processor that stores programinstructions executable by the processor to: display a 3-dimensional(3D) stereoscopic scene via the at least one stereoscopic displaydevice; display the cursor via the at least one stereoscopic displaydevice concurrent with display of the 3D stereoscopic scene; wherein inresponse to the current position being inside a specified volumeproximate to the at least one stereoscopic display device, the programinstructions are further executable by the processor to display a 2dimensional (2D) scene via the at least one stereoscopic display deviceconcurrent with display of the 3D stereoscopic scene, wherein at least aportion of the 2D scene overlaps at least a portion of the 3Dstereoscopic scene, wherein the cursor operates in a 2D mode, whereinthe entire 2D scene is displayed within a plane of a screen of the atleast one stereoscopic display device, wherein the specified volume isdefined by a plane parallel to a screen of the at least one stereoscopicdisplay device, wherein the plane is a specified non-zero distance fromthe screen, wherein the specified volume is on the screen side of theplane, wherein boundaries of the specified volume are defined byboundaries of the at least one stereoscopic display device, and wherein,in the 2D mode, the cursor is usable to interact with the 2D scene; andwherein in response to the current position being outside the specifiedvolume proximate to the at least one stereoscopic display device, theprogram instructions are further executable by the processor to removedisplay of the 2D scene, wherein the cursor operates in a 3D mode,wherein, in the 3D mode, the cursor is usable to interact with the 3Dstereoscopic scene.
 13. The system of claim 12, wherein the specifiedvolume defines a first region and space outside the specified volumedefines a second region, and wherein the second region is displayed asinactive when the cursor is in the first region.
 14. The system of claim12, wherein the 2D scene comprises a graphical user interface (GUI) thatincludes one or more 2D user interface (UI) elements, and wherein, inbeing usable to interact with the 2D scene, the cursor is usable tointeract with the one or more 2D UI elements.
 15. The system of claim14, wherein the specified volume is further defined by respectivebounding volumes proximate to each of one or more 2D user interface (UI)elements on the at least one stereoscopic display device.
 16. The systemof claim 12, wherein the specified volume is further defined byexclusion of a respective bounding volume of at least one object in the3D stereoscopic scene.
 17. The system of claim 12, wherein the 3Dstereoscopic scene includes at least one 3D graphical object; andwherein, in being usable to interact with the 3D stereoscopic scene, thecursor is usable to interact with the at least one 3D graphical object.18. The system of claim 12, wherein in being based on the position of a6 degree of freedom (6DOF) 3D pointing device, the current position ofthe cursor is based on a virtual beam extending from the position of the6DOF 3D pointing device.
 19. The system of claim 18, wherein the virtualbeam has a configurable length.
 20. The system of claim 18, wherein adistal end of the virtual beam comprises a virtual stylus tip, andwherein the cursor corresponds to the virtual stylus tip.
 21. The systemof claim 12, wherein the 6DOF 3D pointing device is a physical stylus,and wherein the cursor corresponds to a tip of the physical stylus. 22.The system of claim 12, wherein in the 2D mode the cursor is displayedin the 2D scene with a first representation, and wherein in the 3D modethe cursor is displayed in the 3D stereoscopic scene with a secondrepresentation, wherein the second representation is different from thefirst representation.
 23. A method, comprising: displaying a3-dimensional (3D) stereoscopic scene via at least one stereoscopicdisplay device; determining a current position of a cursor with respectto the at least one stereoscopic display device based on a position of a6 degree of freedom (6DOF) 3D pointing device with respect to the atleast one stereoscopic display device; and displaying the cursor via theat least one stereoscopic display device concurrent with display of the3D stereoscopic scene; wherein in response to the current position beinginside a specified volume proximate to the at least one stereoscopicdisplay device, the method further comprises displaying a 2 dimensional(2D) scene via the at least one stereoscopic display device concurrentwith display of the 3D stereoscopic scene, wherein at least a portion ofthe 2D scene overlaps at least a portion of the 3D stereoscopic scene,wherein the cursor operates in a 2D mode, wherein the entire 2D scene isdisplayed within a plane of a screen of the at least one stereoscopicdisplay device, wherein the specified volume is defined by a planeparallel to a screen of the at least one stereoscopic display device,wherein the plane is a specified non-zero distance from the screen,wherein the specified volume is on the screen side of the plane, whereinboundaries of the specified volume are defined by boundaries of the atleast one stereoscopic display device, and wherein, in the 2D mode, thecursor is usable to interact with the 2D scene; and wherein in responseto the current position being outside the specified volume proximate tothe at least one stereoscopic display device, the method furthercomprises removing display of the 2D scene, wherein the cursor operatesin a 3D mode, wherein, in the 3D mode, the cursor is usable to interactwith the 3D stereoscopic scene.
 24. A method comprising: presenting a3-dimensional (3D) scene by at least one display, wherein saidpresenting the 3D scene comprises displaying at least one stereoscopicimage of the 3D scene by the at least one display, wherein the 3D sceneis presented according to a first viewpoint with respect to the at leastone display, and wherein the 3D scene correlates to a physical openspace; wherein the first viewpoint has a first viewpoint X, Y, and Zlocation and a first pitch, yaw and roll orientation when referenced tothe at least one display; wherein the 3D scene includes a rendering ofat least one object at least partially visible from the first viewpointwithin the 3D scene, where the 3D scene includes at least one 3D objectrendered in stereo 3D; determining a first virtual viewpoint within the3D scene, wherein the first virtual viewpoint is different than thefirst viewpoint, wherein the first virtual view point corresponds to afirst X, Y, and Z location and a first pitch, yaw and roll orientationin physical open space and maps to a first coordinate in the 3D scene,wherein the first coordinate comprises a second X, Y, and Z location inthe 3D scene, and wherein the first coordinate comprises a second pitch,yaw and roll orientation in the 3D scene; determining a first region anda second region, wherein the first region correlates to the 3D scene,wherein the second region is outside the 3D scene, and wherein thesecond region is bounded by a plane parallel to a screen of the at leastone display, wherein the plane is a specified non-zero distance from thescreen, wherein the specified volume is on the screen side of the plane,and wherein boundaries of the second region are defined by boundaries ofthe at least one display; presenting a 2D scene comprised in the secondregion and a cursor in a 2D operational mode when the determined firstvirtual viewpoint is determined to be in the second region, wherein atleast a portion of the 2D scene overlaps at least a portion of the 3Dscene; and presenting the cursor in a 3D appearance and position andremoving the 2D scene from the presentation when the determined firstvirtual viewpoint is determined to be in the first region.
 25. Themethod of claim 24, wherein the 2D scene comprises a graphical userinterface (GUI) that includes one or more 2D user interface (UI)elements, and wherein the cursor in the 2D operational mode is usable tointeract with the one or more 2D UI elements at zero parallax.
 26. Themethod of claim 24, wherein the 3D scene includes at least one 3Dgraphical object; and wherein the cursor in the 3D operational mode isusable to interact with the at least one 3D graphical object at zero,positive, or negative parallax.
 27. The method of claim 23, wherein thespecified volume defines a first region and space outside the specifiedvolume defines a second region, and wherein the second region isdisplayed as inactive when the cursor is in the first region.
 28. Themethod of claim 23, wherein the 2D scene comprises a graphical userinterface (GUI) that includes one or more 2D user interface (UI)elements, and wherein, in being usable to interact with the 2D scene,the cursor is usable to interact with the one or more 2D UI elements.